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ABSTRACT 

This presentation identifies one of four kinds of 
evaluation — impact evaluation — existing today for career education 
programs. Impact evaluation provides four essential sets of 
information: (1) all data necessary to determine if a program should 
be continued; (2) a determination of which alternative program 
achieves the greatest gains at a given cost; (3) information on the 
components of each program and the component mixes which are most 
effective in a given expenditure to achieve maximum operating 
efficiency; and (4) data from the above three for persons with 
different characteristics so a decision-maker can determine which 
individuals are best served by each program. (NH) 



o 

ERIC 



IMPACT EV. .UATION OF CAREER EDUCATION 

PROG«AMS^- 



STATED D?V,5J\°%^'f * OB OPINIONS 

EDucATioS ;i!,r.;:V^E0P 



Ying C. Chuang, Ph.D. 
Center for Urban Education 

105 Madison Avenue 
New York, New York 10016 



♦Presentation nnade at the 42nd National Meeting of Operations 
Bescarch Society of America, joining with the 1972 National 
Meetings of the Institute of Managennent Science and the System 
Engineering Group of American Institute of Industrial Engineer- 
ing, Atlantic City, New Jersey, November 1972, 



« 

IMl'ACT KVAJ^UAIION OF CAREER EDUCATION PPOGRAMS 

Abstract 

The main thrust of career education programs which have 
received a top priority in the U.S. Office of Education, is to pre- 
pare all students for a successful life work by increasing their 
alternatives for occupational choice, by eliminating barriers to 
attaining job skill, and by enhancing learning achievement in all 
subject areas at all levels of education. The need of effective eval- 
uation of career education programs is acute when large public or 
private resources are involved. This paper identifies one of the 
four kinds of evaluation existing today, impact evaluation of ca- 
reer education programs which should provide at least four es- 
sential sets of information. First, they should provide all of the 
data necessary to determine if a particular program should be 
continued. Second, they should determine which of the alterna- 
tive programs achieves the greatest gains for a given cost. Third, 
evaluation should present information on the components of each 
program and mixes of components which are most effective for a 
given expenditure so that maximum operating efficiency can be 
achieved. Finally, evaluations should provide the first three types 
of information for persons with different characteristics so that a 
decision maker may determine which individuals are best served 
by each program. 
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A fundamental purpose of education is to prepare stu- 
dents to be a productive member of so,ciety. So far, our school 
systems do not perform their function properly. Too many youths 
leave schools without having developed marketable skills, the 
ability to make reasonable career choices, or the capacity to 
attain maximum personal fulfillment from their lives. 

Indeed, careful analysis of the structure and function 
of our present school system compels one to seriously enter- 
tain the idea that it is impossible for the scl :>ol system to ac- 
complish what needs to be done. The present system simply is 
not built to do the job. Efforts continue in many places to reform 
and revise the present educational system. Many efforts have 
met with some degree of success; most offer only piecemeal 
remedies. 

What is required, in essence, is a total educational 
system reform which would strive to provide each student with 
a means to select and pursue a life's path which would maximize 
his aspirations and abilities. The student would have the oppor- 
tunity to develop the kinds of knowledge, understanding, and skills 
needed to live in a work- oriented society. Such a model of educa- 
tion can be considered Career Education. 
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Carc*er Ed uc^.Ui)n 

Under the career education concept, in addition to learn- 
ing how to read, write, and compute, the career education student 
in the elementary grades, I through 6, studies history, languages, 
and the physical and social sciences^ (See Chart 1 for an example 
of a Career Education model). Simultaneously, he explores the 
world of work through a wide spectrum of occupational '^clusters. " 

In the middle grades, 7 through 9i the student examines 
more closely those clusters in which he is most interested* By 
the end of the 10th grade he develops elementary job entry skills 
which he can pursue if he does not complete the 12th grade. If 
he does complete the 12th grade, the student is prepared to enter 
the world of work or to continue his education at a postsecondary 
institution college, technical institute, or other choice suit- 
able to his needs, interests, and abilities. 

All students have the opportunity to enjoy work during 
their school years. This is accomplished through cooperative 
arrangements with school, employer, community and govern- 
ment agencies. Extensive guidance and counseling activities 
assist the student to discover and develop his particular inter- 
ests and abilities and match them against potential careers. 
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Impact Kvaluation o£ Career Educatiuii Programs, pnac threo 

As an oxprriiiKratal alturnativt: to tradJtitmal elemen- 
tary and secondary education, career education han been con- 
ceptualized in three separate forms: career education based in 
the schools, career education bas.»d in business and industry, 
and career education based in the home. The school<*based car- 
reer education idea revolves around the schools's providing a 
greater emphasis than at present on the students' development 
of occupational skills in the course of his high school education. 
Employer-based career education involves business, industry, 
and public and private agencies in preparing youth at the sec- 
ondary level to seek immediate employment on one hand, and to 
continue education to eventual employment on the other hand. 
The home -based career education concept involves the use of 
various media, primarily television, to introduce vocational 
development concepts into the home. 

Need for Effective Evaluatio n 

The need for effective evaluation is nowhere more acute 
than in the field of Career Education Programs where large amounts 
of public and private resources are involved and when the goals of 
the programs may be far -*r cachings 2* 5 million students leave 
the formal education system each year without adequate prepara- 
tion for a career* (Marland 1972)* To the extent that career edu«* 
cation programs have been inefficent or ineffective^ not only arc 
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dollars wasted, but tlu- Htiulcniy do not achieve their full poten- 
tial. 

Past evaluations of educational programs have taken 
a variety of forms. Three basic types of evaluation can be idcn- 
tificd* The first type of evaluation is input evaluation. In con- 
ducting an input evaluation, an attempt ib made to determine the 
r^jicvance of the project to the xieeds of students, community, exn- 
ployers and government. The second type of ov^aluation is process 
evaluation. The purpose of process evaluation is to monitor the 
project operating efficiciency# Project monitoring begins with 
a basic program plan which describes how a particular career edu- 
cation program should bo administered. This plan should include 
euch itejns as the qualifications of the program staff, the administra** 
tivc hicrachy which should be present, the reporting forms which 
should be used, and the list of services which should be performed. 
Project monitoring then will determine if the plan is being followed 
and to what extent procedures and practices may be modified to 
follow more closely the plan of operating efficiency. The third 
type of evaluation is output evaluation. The output evaluation seeks 
to determine to what extent the stated short-run objectives have 
boon accomplished. This type of evaluation arises out of the need 
by program administration for immediate information on the success 
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or failure ul u particular proj ram. Tin: criloria of program suc- 
cess includu such objectives as th« learning of now «Kills, the 
placement record of the project immediately following its com- 
pletion and the improvement in earning of the program partici- 
pants as compared with their earnings before the program. Most 
of the cvaluators of socially oriented prograir^s carried on in the 
past have been these three types of evaluations. 

Impact Evaluation 

The fourth type of evaluation is impact evaluation or 
outcome evaluation. The impact evaluation examines the long- 
run objective of career education programs and views success 
and failures in those terms. The major purpose of this type of 
evaluation is to provide policy makers on a government or other 
funding agency level with basic data necessary for them to make 
decisions wisely, Lnpact evaluation of career education programs 
should provide at least four essential sets of information. First, 
they should provide all of the data necessary to determine if a 
particular career program should be continued. Second, they 
should determine which of the alternative programs achieve the 
greatest gains for a given cost. Third, evaluations should pre- 
sent information on the components of each program and the mixes 
of components which are most effective for a given expenditure so 
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that maximum operating efficiency crun bo achicvcfh Fourth, 
evaluations sliuuld pruviUu the: above three lypKi& q£ information 
for students with different characteristics, so that a decision- 
maker may determine which individuals are best served by each 
program. 

There are four parties who may benefit from career 
education programs. These arc students in the program, society 
as a whole, employers, and government. Each of these groups 
has different outcomes which they wish the career program to 
accomplish. Therefore, the goals of career education programs 
will differ. For the student, the goals ere more limited and are 
usually those which directly affect him, such as receiving a 
high school diploma, getting a job, attending a technical school 
or a four-year college. From society's point of view, the goals 
of career education programs are increased aggregate production, 
improved equity in the overall distribution of income and employ- 
ment, and reduction in the national unemployment rate. An em- 
ployer will tend to look at the programs in terms of his interest. 
For instance, ho will be concerned about how the productivity of 
his labor forces has been increased. Finally, the government will 
view the program in terms of the various social goals and, in 
addition,, will seek programs which will help its party stay in 
power and aid its budgetary position. 
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Obvit)u^*ly, ihcro is> a yroat deal of overlap among the 
goali^ of the four groupK* The governnuint acrt s as the agent of 
society in operating the programs* As such, definitions of pro- 
gram success will naturally coiUcide in most areas for the gov-^ 
ornmcnt and for society. Similarly, students and employers as 
members of society are interested in aggregate changes as well 
as those directly affecting them. Likewise, the effect of pro- 
grams on students will determine in part their success in terms 
of society^ Increased employment of students in programs is 
likely to improve aggregate employment, and improvement in 
the production of individual firms may lead to increased aggre- 
gate production. 

There may also be an overlap between the goals for 
each of the parties* For instance, the reduction in an individu- 
al's unemployment may increase his earnings as well as decrease 
his feeling of dependency* Since the effects, however, may have 
independent importance for the individual, we believe that all 
should be considered* 

There also may be conflicts, however, between the 
goals of the different parties and between various goals for a 
particular party* Thus, we may find that a program which im- 
proves the income of the participants is very costly to the 
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government because a program wliich is highly efficient at in- 
creasing aggregate production leads to greater iuequity in the 
national distribution of inc omu* These conflicts in possible pro- 
gram achievement raise the problem of ranking the goals of each 
of the parties. On a theoretical level the cvaluator should recog- 
nize that the rc^vards and costs of career education programs to 
particular interested parties who have political influence may 
play an important role in determining the sixe, scope, and even 
the existence of the program. He must then take account of the 
goals of all four parties. 

Furthermore, it is necessary for the career education 
program cvaluator to present data on many program goals so that 
the different parties will have the data necessary for them to eval- 
uate the program. Ideally, the users of career education program 
evaluations should specify those goals which they believe to be the 
most important. The cvaluator and decision-makers v cald consult 
each other to ensure <:hat the evaluation measured the most mean- 
ingful goals. In addition, the relative ranking of the goals may 
change over time. Thus we suggest that the e valuator examine as 
many goals as he can in his study. 

To facilitate the choice of go:Us to be studied, we pre- 
sent a list of goals for students, society, the individual employers, 
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and the j»ovcrnx:u;nt» Wo i< vA Uial all career education programs 
can be judged to a large extent in terms of these goals. We rea- 
lize, however, that each program will have a different method 
of reaching these goals and will put somewhat different empha- 
sis on each of the goals. The list is, of course, not all-inclu- 
sive. It should, however, provide many of the most important 
goals of career education programs. Below each goal we pre- 
sent specific operational criteria to measure the success of a 
career education program in meeting the goals. 

This list does not assign priorities to specific goals. 
It is believed that assignment of priorities is the ultimate res- 
ponsibility of the decision-maker. For each of the possible de- 
cision-makers two types of goals have been presented. The 
first group includes goals where operational criteria exist and 
are presently being used for evaluation purposes. The second 
set of goals are included with the hope that further development 
of operational measures will be stimulated. 

I. Career education program for students 

A. Objective with clear evaluation criteria 

1. Received a job 

2. Admitted to a four-year or two-year college 

3. Admitted to a technical school for further 

technical training 
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4. Received a hijjh jichool dip]i>ma 
h. Objective more difficult to ruea«ure 

U Increased satisfaction with school work 

2. Increased satisfaction with social status 

3. Improved self-concept 

4. Increased personal income 
li. Career education program for society 

A. Objectives with clear evaluation criteria 
U Reduced crime 
2» Reduced unemployement 

3» Improved equality in distribution of income 
and employment, especially for target 
group 

Bm Objectives more difficult to measure 
U Increased social satisfaction 
2^ Increased voluntary leisure 

3. Improved family life 

4. Stable consumer price 

5. Improve race relations 
III* Career Education Program for employers 

A» Objectives with clear evaluation criteria 
U Job of specific employer filled 
Z. Job in particular area filled 
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3. lucrcasocl sales 
B# Objoclivos mni*o difticult to mcasuro 
U Increased job satisf«i ction 
Z. Improved profits 

3, Increased production 

4. Improved employer-employee relations 
IV. Career Education Program for government 

A. Objectives with clear evaluation criteria 

1. Increased registration for a given political 
party 

Z. R educed welfare receipts 

3. Reduced unemployment insurance 

4. Reduced cost of government operations 

5. Increased tax revenues through an increased 

tax base 

B. Objectives more difficult to measure 

I. Improved national health 
Z. Increased voluntary leisure 

3. Improved housing conditions 

4. Increased national production 

Measuring Profiram Success 

Because benefits of career education programs may be 
received by individuals, by society, by employers, and by the 



Impact K valuation oi Career Education Programs, pauc Iwolve 

govcrmncnt, and bccaussc oatli h;i« rfonunvbai ciitKtrunt ^oaLs, or 
criteria of success, separate calculations of program bcxuifits 
are required for each of these groups on the basis of the bene- 
fits as defined for that group. The methodology for each calcula- 
tion will be approximately the same for all of the groups, how- 
ever. Each group seeks to determine the differences which exist 
in a set of measures, with and without the career education program 
This will require a comparison between the experience, behavior, 
and attitudes of program participants after participating in the 
program and those expected if they had not participated. 

To conduct an evaluatio«i of a career education program 
it is necessary to measure the relationships between the program 
goals (the dependent variables) and a variety of independent varia- 
bles including the personal characteristics of participants, the 
program components, and the conditions under which the programs 
operate. It will be the job of the evaluation to discover which of 
these independent variables are important and the nature of the 
relationship. However, most dependent variables with which 
evaluations of career education programs deal are functions of 
more than one independent variable. Under these circumstances 
the analyses would treat simultaneously all of the independent 
variables which are believed to be relevant. To omit some vari- 
ables in the analysis may lead to distorted conclusions due to 
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corrcJaticni •.•racUoii ..i>i)ong Ihi'.si.: variablos and iliosit: indo- 

pendent variabl-js which arc included in the analysis, Thorelorc, 
multivariate techniques should be used in the evaluations to dis- 
cover and test the statistical significance of any rclalionship ^ 
which are observed. 

The use of simple cross tabulations to isolate such 
relationships will be inadequate in most cases. For instance, 
the effects of race, age, education, and skill level on earnings 
are all interrelated. Yet each of the effects should be distin- 
guished. To cross tabulate by all of these variables would in- 
volve so many cells that the sample would have to be enormous. 
In addition, the lablr s would be so large as to be unmanageable. 
Multiple regrt b . ic.n and currrlation techniques, on the other 
hand, recjuin.- <i^:\ic}i smaller sample si/.o and permit easy 
interpretation of the findings. 

The costs of a career education program most properly 
should be considered to be the program's opportunity cost the 
value of the alternative benefits which are foregone because of the 
program. Resources which are devoted to the career education 
program cannot be used to produce other goods and services. For 
instance, society by devoting manpower to conduct training programs 
loses the services of those persons as teachers in vocational high 
schools, as stock brokers, or even as automobile workers, to 
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use a fuvv i:X*Mnp)t»s, Auothi:!* aUcrnntiw is ihixi th«N^;c p^rsonn wou 
be uncmployc<l, in which case, society gives up nothing in h>st pro- 
duction by putting them to work^ Simihxrly, the government gives 
up altcrnalivc programs or tax cuts and employers give up plant 
improvements or dividends when they spend funds for career edu- 
cation programs* Finally, individuals may lose earnings while 
they participate in the program. 

The costs of career education programs can be viewed 
from several different perspectives, just as wore their benefits* 
As explained above, students, society, employers, and govern- 
ment may each be required to give up resources for use in the 
programs. In some cases expenditures of resources will mean 
lost opportunities for more than one group. For example, sala- 
ries of government administrators will be costs for society as 
well as for government. There will also be expenditures, how- 
ever, which will be costs for one group but will be gains for 
other groups. For example, government allowance payments to 
program participants or reimbursements paid to employers will 
be costs for the government but will actually reduce the costs of 
the participants and the firms involved. Therefore, we once 
again present separate lists for each group. 
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!• Coat for S{ucK:nl:i 

U The cost which stuclont.s would not need if they do 
not participate in the program* These include such costs as: 
transportation to and froii) the prograDi, rnealh and living costs 
away froiii home, uniforms, books, tools or other educational ma- 
terials and day care for dependents. 

2, Loss of earnin^js due to participation in the pro- 
gram. 

3» Reduced govermnent payments due to participation 
in the program. This includes training allowance, subsistence 
allowance, travel allowance or unemployment insurance pay- 
ments. 

II. Cost for Society 

1. The cost of all personnel involved in the program. 
This includes local project personnel at the state level and at 
the national level. 

2. The physical capital used in the program. This 

includes: 

(a) The market rental value of all property 
and building including government property. 

(b) The market rental value of all equipment 
and materials used in the program. 
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3» MibCi:lKirKM>us s^o rvjc<.ti.i ll^rCiJ^^;ary 1i> operate thu 
program; such as staff travel^ telephone service: and oquiimient 
repair* 

4. The potential procluctioii of persons participating 
in the program which is lost during the time the program is be- 
ing conducted, 

III* Cost for Employers 

I. The wage costs of cmph:>yecs who perform services 
minus any funds received from the government to reimburse the 
firm* 

2« The vahxc of all physical capital used up in the pro- 
gram which is owned by the firm and for which it docs not receive 
reimbursement from the government, 

3* Miscellaneous expenditures by the firm on services 
necessary for program operations which are not reimbursed b^ 
the government* 

4* The production lost because of poor quality of work 
which results as a part of the learning process for the participants* 
From this cost any reductions in taxes as a result of lower profits 
and any reimbursement from the government should be subtracted* 
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IV, Cost for Guvcrimu'Ut 

1. The personnel eosts of al) personnel involved in 
the program for whom the governmenl pays the salaries or re- 
imburses loeal sponsors. 

2. The value of all physical capital used in the program 
which is government owned, rented by government, or for which 

a local sponsor is reimbursed. 

3. The expenditures on miscellaneous services which 
arc made by government or for which payment is reimbursed by 
government. 

4. The net increase in government payments to individu- 
als which are made to induce them to participate in the programs. 

5. The tax revenues which are lost during the pro- 
gram. These would include tlie reductions in the personal income 
taxes, social security taxes and unemployment insurance taxes 
which may result from lower earnings, and the reductions in 
sales and excise taxes caused by lower expenditures of the partici- 
pants while they are in the program. In addition, lower corporate 
profits taxes might result from decreased efficiency of partici- 
pants involved in on-the-job programs. 

6. The other items for which government makes pay- 
ments to local sponsors. These would include payments to firms 

conducting on-the-job training to compensate for the lower pro- 
ductivity of trainees. 

ERIC 



To xiiea.sure tlio costs of *\ program rocjuircs the com- 
parison of the opportunity cost or thci expenditure of resources un 
behalf of the program participaaif^ by IhemscUes, by society, by 
employers or by government, with those which would have occurred 
were tliere no program* Wo should nute again that while we desire 
to measure the effects of adding or subtracting participants from 
the program, we usually are unable to do this* Instead we measure 
the average costs for a program and must assume that the program 
with higher average costs will have a higher cost for adding a new 
participant. ^ 

As was the case with measurements of program success, 
the best way to ineasure wlxat would huve happened to the program 
participants is to use a control group which is randomly selected 
from persons willing and able to enter the program. Again, only 
this group will give an unbiased estimate* Thus if costs are to 
be accurately estimated, the saine type of control group must be 
used to measure them as is used to measure program success* 
If projects are selected for evaluation when funded, this will per- 
mit the same control groups to be used to measure the costs and 
success of a program* 
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1. TilL UiiECl' CONilUJJ. c;UO Ul'.'i. Control groups 
should bo used to provide information for thriu; types of cost 
estimates. The first is the Iossch incurred while the partici- 
pants are in tlie career education program (the opportunity costw). 
While particii)at.ion in the program, individuals uaually are not 
engaged in what they normally would be doing. Therefore, par- 
ticipation in the program may lead to losses of after tax earnings, 
unemployment compensation, or welfare payments by the individ- 
uals, production by society, and taxes by government. The experi- 
ence of the control group during the course, however, should not 
be affected by the program. Thervfore, the difference between 
their after tax earning;?, unemployment compen.sdtlon, welfare 
payments, production and taxes, and those of the program parti- 
cipants will show the losses actually incurred because of partici- 
pation in the program. 

The second use of control groups is to determine how 
much of the governmental services received by the participants 
would not have been received if there were no program. Earlier 
we discussed employment service job referral services which arc 
normally used by many of the persons who enter career education 
programs. Similarly^ when welfare recipients enter career educa- 
tion programs the counseling they receive in the program may merely 
replace counseling they would have received from a case worker. 
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Therefore it is iiupi>rtaiii ihat infinMuatJint hv i nllcctird on tlut 
amount and n;iiurc uf all govi^rnnii^ntal services received by 
both the participants and the control group* If this is known, 
the latter can be subtracted from the former to find the actual 
incrcmcxit in sorvicoK which result from a program* Then only 
the cost of this increment in services should ]}o compared with 
the benefits which were calculated as the differences between the 
two groups. 

Finally, the control group can be used to measure the 
increment in program- related expenditures by the participants. 
Some programs require the participants to incur (expenses for tra- 
vel, instructional matcri»ils, uniforrr.s, living expenses and meals 
away from home, etc. Some of these expenditures represent 
added costs of program participation. Others, however, may 
not. For example, if an individual would be taking the bus to 
work instead of taking it to a career education center were he 
not in a career education program, there may be no additional 
cost of transportation resulting from the program. To arrive 
at this conclusion, however, it is necessary to know the expend* 
ituros associated with the course by the participants and the ex- 
penditures on these items by the control group. 
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I. TliK UtiF. or MIJ LriVAKIAT]:: ANALYSTS, la uur (\is- 
cussion of the measurement of program succors \vc indicated that 
multivariate analysis sJiould bo used to determine how the per- 
sonal chnracturisticto^ the program charactcribtic^i, and other 
facinvB aiiccted each of the goalst The same procc.d\ii*es should 
be used to examine costs. Only if this is done will it be possible 
to relate differing program success which results from changing 
the nature of the program with the costs for making these changes. 
For example, if the job development component is found to be 
twice as effective as tlic counseling component, one must also 
know the relative costs of the two sets of components before any 
changes are introduced. Thus the costs for society, participants, 
employers, and government should each be analyzed using the 
same independent variables as we discussed in the section on 
benefits 

The multivariate analysis of costs, however, can be con- 
ducted on two levels* The first is to examine total costs of pro«* 
jccts. Each project examined in the study is an observation and 
total project cost, total project cost per enrolled participants 
(studcnts)> total project cost per graduate participant^ or total 
project cost per hour of student participation can be the dependent 
variables. Such an analysis could tell the effect on total costs of 



the: Ciiniponcnlii^ whicb. inahc up \hc pru[u**^nu 'ihi^i is thu t:yj>o f)f 
average cost infurnjatiou vvliich is usually dc^siritd* If, however, 
it was u^rcful 1u know the effect of siicl; chang^^B (Mi thv. costH of 
particular cuxnpL;uents of the prograui then the dopendont variable 
would have to be only the cost of tliat compoiient, the cost of the 
coaiponv.i^t per enrolled participant or gi-aduale participant or 
the cost of the conipoj*ent per hour of participation* 

Another appro<>ch wumld be to trc.at e ach participant 
in the program as the unit of observation, Thib type of analysis 
is analogous to that ui>ed to measure program success and could 
also providi? more detailed information because iJiere would be 
a greater number of obr^urvations. It is, howewr, nnich more 
difficult to conduct* 

The data required, though, would bo the same as those 
which wc felt necessary to estimate the effect of changes in pro- 
gram components. Each individual participant would need a 
form on which would be entered all services performed in the 
program for the individual, the time spent in providing these 
services and the identity of the person who provides tliem. The 
costs of the services would then be computed by multiplying the 
hourly rate of each person providing services by the amount of 
time provided* To this would be added a figure representing 
some apportionment of the administrative costs, tlae costs of 
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capilTil ♦:iic:litirh. and thi' cost m" i.tist cllarn'onf. t;ovvic<:s. Tlio 
apportionnu:nt might ho based on ihv. length of tlmo the participant 
was in tho proi;ram. Finally, the individual's additional cxpcndi- 
turcs and opportunity costs of hr-ing in ihc program would bo addod. 
Those individual costs would bo the depondont variable with the 
same independent variables as discussed oarlior. 

Conclusion 

The data gathered for evaluation of career education 
programs should provide the information to make four types of 
decisions: (1) whether a particular existing program should be 
continued, {2) which of several alternative e.vi sting programs 
should be expanded or contracted, (3) in what vays can changes 
in the components of a particular program lead to improved ^iffi- 
ciency, and (4) for particular groups of individuals, what pro- 
grams serve them best? 

Although the data discussed above could answer these 
questions, a criteria of combining the measures of program 
success and cost may be important to a decision-maker evalu- 
ating a program. In detail: 

SHOULD A PROGRAM BE CONTIiNUKD? The answer 
to this question usually depends on what alternative programs are 
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ciViiilabl V. Ciu'h 'r.M a^;r vcMu»ni , hfjwt'Vt'V, .should exist im ihv dis- 
continuation of certain typt^s of progiNtms. A program Jihould bu 
discontinued wh^n no rodccniing features arc found aftc»r consider- 
ation of all crit^^ria of !^uccesi>, that is, where all important de- 
pendent variablcfi are measured and: (1) no benefit- cost ratio 
is greater than one, and (2) no cost-cifcctivcnuss ratio has a 
positive numerator or a negative denominator^ 

These criteria will very seldom bo met if only because 
it will usually be i ipossible to quantify all of tlic dependent varia- 
bles* Therefore, the program decisions must be based on com- 
parisons of alternative programs. % 

COiMPARlSON OF ALTERNATIVE PROGRAMS. In a 
very few cases one program will be superior to another program 
when compared on all of the criteria we have suggested. In these 
cases the course of action is clear: the superior program should 
be expanded. (This is based on the assumption, discussed earlier, 
that average benefits and costs are positively related to those at 
the margin)* In most cases, however, one program will be 
superior in some areas but inferior in others. The choice of 
program expansion and contraction under these circumstances 
depends on the preferences attached to each of the goals* For 
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o>;au;plc> a .skill tvaijui;-; pri)^»raia liiay mure ;:i7c:cUvi: tluxu a 
career oritnitation prngram in raisinj.; the oaruiags and rutluciitg 
the unumploynuuit of iho participants. The career orientation 
course, however, n^-iy lead to greater persona] satisfaction and 
improvement in future career decisions, Li this situation, 
assuming that only one program can be expanded, a choice must 
be made as to which is more important, increased earnings and 
employment or behavioral improvements. Once explicit weights 
showing relative importance arc assigned tu each of these goals, 
the program decisions can be made. The weights should be expli- 
cit So that others who have different values can also use tlic anal- 
ysis. 

As we discussed earlier, there are two strategies 
which may be followed in assigning relative weights to program 
goals. The first is for the decision-maker to provide the evalu- 
ator with the weights of various goals before the evaluation is 
begun. The cvaluator will then examine only those measures of 
success with non-zero weights and will igatc his findings to 

arrive at a single overall measure of program effectiveness. 
The advantage of this approach is that it does not consider what 
are thought to be irrelevant goals, that is, those given no weight 
by the decision-maker, and so is more economical and efficient. 
Its mcijor shortcoming is that the weights assigned to goals differ 
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anuiav^ Uici;^i;jn inalvirr.s aiitl ovur tiixii^s 3'^r*x* ihl.^' rcvxt inx Llio al - 
tcrnativo approach uHuall/ is more practical. 

The second strato^^y prupusus that the cvaluator should 
calculate thw benefit- cudt or co.st- offcctivitness ratio for every 
goal which nii^»l>t bo rtjlevant for eacli program being examined. 
1£ considertition of all possible goals is not possible because of 
cost or other liiuitations, then the calculations should be made at 
least for all goals which are thought might be highly relevant. The 
ratios for alternative programs can then be compared in a single 
table. This procedure allows each decision-maker to assign the 
weights he believes are most appr^opriate and to arrive at a de- 
cision of overall program vaUie. If circvmnstances change, the 
decision-maker can rcdefin<r the weights he wishes to use and 
simply recalculate the relative performance of the programs. 
The vv'sights should be determined independently of the analysis 
results, however. Otherwise there is a great post-analysis temp- 
tation to find the weights which will make the analytical results 
conform to previous prejudices. 

COMPARISONS OF PROGRAM COMPONENTS AND 
OF THE DIFFERENT GROUPS OF PARTICIPANTS. The same 
procedures could bo used to compare the successes and costs of 
the components of a particular program. The multivariate anal- 
ysis proposed includes the effects of the presence, the duration, 



Pof^sibly^ llicre may a cumponcwit which has no bcnu- 
fit-cost i*ai h) ^riNt.tcr than om- and no po«ili\^« ci)st-cffcctivoncss« 
ratio foi' all possible critoria. Such a component probably should 
be druppod* In sotno ca^c:s, huv/i vcr, compunonts have to bo 
treated a^ suts. For oxampK., diganootic testing by itsoU will 
make no improvument in the individual's behavior* Without it, 
however, u;5cful career counseling may be extremely difficult. 
More likely, however, components will vary in their effective- 
ness depending on the criterion of success. Once morc^ a tabular 
listing for each con^poac^nt can be mad*.t of the benefit-cost or 
cost-effectiveness ratio for each of the criterion to facilitate 
the choice between components. 

Finally, the same method of analysis and presentation 
could not be used to identify the effects of different programs and 
components on different types of participants. The multivairiate 
an alysis would show wiicthcr programs or components produce 
differential success or costs depending on the types of partici- 
pants* From these data benefit-cost and cost-effectiveness ratios 
for a particular group of participants could bo calculated for all 
programs and components. Once more, the weighting of tabu- 
larly presented values will allow cross -program and cross-com- 
ponent comparisons. 
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